In a method for obtaining an external parameter (position and orientation) of camera for image sensing of actual space, methods other than utilization of obtained video image, e.g. measurement of position and orientation of camera by using a position and orientation sensor such as a magnetic sensor, are known. However, in such measurement by position and orientation sensor, the accuracy of measurement is insufficient. Accordingly, if a virtual object is overlay-displayed in actual space based on the position and orientation of the camera measured by the position and orientation sensor, the object may be shifted from a desired position.
Conventionally, several methods for improvement in accuracy of position and orientation of camera for image sensing of actual space have been proposed. One of these methods is providing feature points (land marks) in already-known three-dimensional positions in actual space, and correcting the position and orientation of the camera, by using the difference between an actual position of a land mark of interest on an image-sensing plane and a predicted position of the land mark of interest on the image-sensing plane based on the position and orientation of camera at this time.
Further, another one of the methods is calculating the difference between an actual position of each land mark of interest and a predicted position of land mark of interest based on the position and orientation of camera, and correcting the difference by averaging, thereby further improving the reliability of correction on external parameter of camera.
However, in any of the conventional position and orientation correction method, the difference in a rotation angle about the visual axis cannot be removed. The visual axis V here is a straight line connecting a principal point L on the image-sensing plane, as one of camera internal parameters, and a view point position (center of lens) C (FIG. 1). As shown in FIG. 1, the principal point L is an intersection between the visual axis and the image-sensing plane, and the view point position C is calculated upon calculation of camera internal parameters by preprocessing which is called camera calibration.
In general position and orientation correction methods, correction is performed by moving the view point position in parallel to the image-sensing plane or rotating a view point orientation in a pan and/or tilt direction. However, in any of these methods, there is no component in the rotational (roll) direction about the visual axis. Accordingly, in the above-described general correction methods, as the difference in the rotational angle about the visual axis cannot be corrected, even though an image of actual space and an image of unrealistic (virtual) space are combined, the respective positions do not accurately correspond with each other.